Mixed reality simulator for professional training

ABSTRACT

The present disclosure relates to computer-based training utilizing simulation, and more specifically to mixed and augmented reality simulation software for professional training purposes, including but not limited to mechanical training.

The present disclosure relates to computer-based training utilizingsimulation, and more specifically to mixed, augmented and virtualreality simulation software for professional training purposes,including but not limited to mechanical training.

Within a mixed-reality environment, a user can interactive withsynthetic actors and virtual objects (hereafter collectively referred toas “virtual objects”). The term “virtual object” is intended to include,but is not limited to, any object that can be rendered on a display. Theterm “mixed-reality environment” is intended to include, but is notlimited to, a visual domain wherein a real world user can interact withvirtual objects integrated into a user's view of the real world.

Techniques are described herein that provide for systems, methods, andnon-transitory computer-readable media for simulation based training.This methodology uses augmented reality, which shall as well be used asa term for mixed reality or virtual reality subsequently, a particularincarnation of virtual reality, in order to greatly advance the degreeof equipment fidelity, environment fidelity, and psychological fidelityavailable in simulation for training, with certain embodiments alsodecreasing the cost and improving portability of simulation systems,depending on the particular hardware utilized to realize the simulationsystem.

Here, an augmented reality or mixed reality environment refers to theperception of a user of their real, physical environment with theaddition of virtual, projected, two or three dimensional objects in thatenvironment. Integral to the concept of an augmented reality or mixedreality environment is the feature of the virtual objects to beperceived to exist in the real space as if they were real objects, withthe ability of users to walk around them and see them from differentangles, as appropriate. In harnessing this for training, the methoddescribed enables a replication of nearly an infinite number ofenvironments.

Processes are later described herein for establishing an augmentedreality environment, utilizing a computer application to authorsimulation scenario cases, processing actions on virtual objects, andrecording the events transpiring while users are immersed in theaugmented reality environment. Such processes are at present costly andtime consuming and one objective of the present application is toovercome said problem.

Such scenario cases may be comprised of instruction sets and associatedmetadata and subordinate instruction sets and data, such as audio dataor image data, and may be distributed locally or broadly through the useof an internet infrastructure, marketplace, or other distributionmechanism. In particular scenario cases for training of employees couldbe distributed within a complete company easily after being establishedonce.

Establishing an augmented reality environment within a real space refersto using computer generated virtual models, or virtual objects,projected into the space, where the virtual objects behave as if theyare physically in the space, and where multiple users can see eachother, or at least placeholders of each other, and the virtual objects,and interact with the virtual objects and each other, mimicking asituation where all virtual objects and users are physically in the realspace and are solid objects or beings, even in the scenario where theuser themselves are not in the same real space.

Virtual objects may be projected into empty physical locations or theymay be projected over existing real objects and partially or fullyobstruct the actual view of the real object while allowing for physicalinteractions, such as touch, with the real object, and allowing for suchinteractions to be detected and the virtual object updated based on theinteraction.

Utilizing a computer application to author scenario cases refers to theuse of an authoring application which outputs an instruction set thatdefines the appearance, properties, and behaviour of virtual objects inan augmented reality environment with respect to time, as well asdefining virtual objects effects on other objects in the augmentedreality and the effect of other objects on the virtual object.

Such authoring applications are quite complicated to use nowadays.Mostly the same could only be used with individual programming skillsand they could not be used by technicians or employees of companiesthemselves to set up training scenarios.

Therefore, one objection of the present disclosure is to ease theauthoring of scenario cases, in particular for training scenarios, usingaugmented or mixed-reality environments. In particular one objective ofthe present disclosure is that no programming skills are needed toauthor a scenario case in such an environment.

The present objective is solved by a method comprising establishing,using at least one computing device, an augmented reality environment;

wherein the augmented reality environment is comprised of a real spacein which a user is physically located;wherein the augmented reality environment comprises a set of virtualobjects projected into the real space assignable to physical objects bythe user;sensing, by at least one sensor, a particular physical action of theuser in the real space in which the user is physically locatedrepresentative for the assignment of the virtual object to the physicalobject;in response to sensing the particular physical action in the real spacein which the user is physically located, updating, by at least oneapplication executing on the at least one computing device, a set of oneor more properties of a list linking the physical and the virtualobjects based on the particular physical action sensed by the at leastone sensor, and wherein updating the set of one or more properties ofthe list linking the physical and the virtual objects based on theparticular physical action sensed by the at least one sensor causes asubsequent user to see in the established augmented reality the virtualobjects assigned to the physical objects by the user.

Thereby, it may be preferred that updating the set of one or moreproperties of the list linking the physical and the virtual objectsbased on the particular physical action sensed by the at least onesensor causes subsequent users to see the virtual objects assigned tothe physical objects.

The subsequent user is the first user interacting with the augmentedreality environment after having assigned at least one virtual object tothe physical object comprising the updated list and/or a new user notidentical with the first user.

The surprising finding of the present invention is that the authoring ofscenario cases, in particular of training scenarios, could be easedsignificantly by setting the same up in the augmented realityenvironment directly itself instead of programming the same in advanced.This makes it possible for everyone to set up scenario cases, evenwithout any programming skills.

To do so, it is according to one embodiment of the present disclosurepossible that a user assigns virtual objects to physical objects byphysical actions. The set of virtual objects may comprise functionalelements, in particular visualizations and/or symbols for physicalactions, preferably visualizations and/or symbols for take, use, placeand/or store a physical object, assembly positions, assembly pathways,different types of warnings, tools to be used, general information,preferably a turning moment, inspection tasks and/or how to click aclip.

It may as well be possible that videos are recorded of movements andaction of the first user upon request of the first user by the computingdevice and assigned to a certain physical object. The video itself maybe a three-dimensional video or a three-dimensional visualization of therecorded scene generated from the computing device with the help ofsoftware known in the state of the art automatically or upon request ofthe first user.

It may be possible that this video is shown afterwards as a further helpto the subsequent user and therefore being part of the trainingscenarios.

To link said virtual objects to real physical objects makes it quitesimple to create a training scenario. The user is enable by the methodaccording to the present disclosure to make a physical movement toassign for example a “take” symbol to a certain tool. Afterwards, the“use” symbol is placed above, next to or directly at the physical objectthe grabbed tool should be used. Such steps could easily be completed byinstructions and warnings as well as general information which all canbe linked with physical objects just by a physical movement of the useronly.

It may be in particular of advantage that the training scenarios arecreated directly in the workshops, factory halls or the like andtherefore directly at the places where the users to be trained will worklater on. It is therefore possible to directly integrate the trainingscenarios into the real working environment of the users.

Thereby, it can be of advantage according to one embodiment of thepresent disclosure that the list linking the physical and the virtualobjects based on the particular physical action is generated in theorder the physical actions are performed by the user and shown in saidorder to the subsequent users.

By sensing the order of the physical actions performed by the firstuser, each subsequent user will work within the established trainingscenario along that list and the system will show the same step by stepfunctional elements representative for general information and/or foractions to be performed.

It is of course possible that the order of said list is re-arranged andchanged later on freely.

One aspect of the present disclosure is thereby that the authoring ofscenario cases is as easy as possible. Example given, a first user wantsto explain how to assemble a fuel tank to subsequent users. The firstuser activates the augmented reality environment and points to a virtualobject e.g. “take”. By moving for example the “take” symbol over thefuel tank, the list is updated and the subsequent users will directlysee in the augmented reality environment that he has to take the fueltank.

The next step could for example be a “walk” instruction to a certainlocation so that it is clear for the subsequent user that he should takethe fuel tank and walk to said location.

It is obvious that this example is not limiting the scope of the presentapplication, there are several alternative known and disclosed in thepresent application how to assign a virtual object to a physical object.This very simple example shall only ease the understanding of thefurther explanations below.

According to one embodiment of the present disclosure, the particularphysical action by the at least one user is comprised of at least onebody movement, direct input of data into the at least one computerdevice, at least one vocalization, or the expression of specific ideasvia body movement or vocalization.

Thereby, the at least one body movement comprises hand movements,viewing at positions, eyetracking and/or the use of interaction devices,in particular the use of a joystick or a controller.

It may thereby be preferred that the particular physical actionscomprise a predefined component to be executed by the user to indicatethe system to assign a virtual object to a physical object. In thealternative, the particular physical actions to assign a virtual objectto a physical object is determined by the at least one computing deviceautomatically, and preferably no predefined movement of the user isneeded, in particular wherein the assignment of the virtual to thephysical object is determined by the at least one computing device bymachine learning and/or artificial intelligence.

Machine learning generally includes the construction or generation ofmachine learning algorithms that can learn from data. These algorithmsare used to build a model based on features that is used to generate aclassifier tuned to a particular purpose. Active machine learning is adiscipline where a “teacher,” such as a user, presents training examplesto train a function to the model.

Historically, whether training examples were labeled or unlabeled hasbeen based on the particular purpose. For example, in existing systemstraining examples for implementing a classifier tuned to classifydocuments about the subject of baseball typically include examples ofdocuments labeled as relating to baseball and examples of documentslabeled as not relating to baseball.

Other existing training examples were unlabeled. For example, unlabeledexamples might or might not have been related to baseball. Accordingly,a third party such as the teacher must label existing unlabeled trainingexamples so that the model has valuable input by which to learn anassociated function.

In particular, active learning necessitates relatively high-qualitylabeled training examples such that the model can adequately learn thedesired function for future classification of any number of unlabeledinput documents. However, the discovery of high-quality labeled trainingexamples amongst the virtually unlimited number of unlabeled documentsavailable to the machine learning algorithm is typically costly. Forexample, many users are employed to interpret unlabeled documents todetermine viability for machine learning purposes. However, if aparticular model being trained by existing machine learning algorithmsneeds to be limited, the viability of each potential candidate for alabeled training example must be even more carefully considered, andcosts can exceed desired targets.

Machine learning in the sense of the present disclosure may bepreferably active machine learning. For example, the first user makes aspecific body movement and the system learns what is meant by this bodymovement with the help of an active feedback of the first user.

The method according to the present disclosure allows several options tolink virtual and physical objects to generate scenario cases. One isthat the system just “learns” automatically what to do. This means thatthe system sense and recognized the actions made by the user and theobjects used by the user and captures the same for showing the same tothe next user. This is of advantage to make quick and fast trainingscenarios without any further interaction with the system. Additioninformation like manuals, general information and warnings could beadded later.

The other possibility according to the present disclosure is that acertain physical action has to be performed by the user for the linkingof objects. In particular, it is of advantage according to oneembodiment of the present disclosure that the virtual objects aredisplayed to a user and could be selected for example with the help ofeyetracking and/or a controller. It is of course as well that virtualobjects are selected via body movement or vocalization or a combinationthereof.

For example it is possible that a user points to a virtual objects,keeps the finger on it an places at or next to the desired physicalobject. By this, both objects are linked. As well, it could be possiblethat the user “talks” with the system by using a special key word as thestarting word for communication and directs the system by speech.

As well, to optimize the training scenarios and/or to make audits it ispreferred according to one embodiment that updating the set of one ormore properties of the list linking the physical and the virtual objectsbased on the particular physical action sensed by the at least onesensor causes the comparing of the sensed physical actions of asubsequent user with the predetermined actions defined by the assignedvirtual objects, wherein similarities and differences are sensed by theat least one sensor and compared by the at least one computing device.

Thereby it may be of advantage that the level of detailing of thevirtual objects assigned to the physical objects shown to a subsequentuser depends on a level of experience of the subsequent user, inparticular determined by the monitored similarities and differences ofexecuted actions and predetermined actions defined by the assignedvirtual objects.

In the first step, a first user authors a training scenario. There is noneed that all information comprised by said training scenario have beenshown to every subsequent user at any time. Maybe after several reviewsa subsequent user knows what to do, or for audit reasons is may be ofhelp to record and compare the skills of a subsequent user by justcomparing the actions with the actions that are predefined with the helpof the virtual objects without showing the virtual objects at all.

As well it is according to one embodiment of advantage that the userviews the augmented reality environment by wearing at least oneapparatus that projects virtual objects into the real space in which theuser is physically located; wherein the at least one apparatus includesthe at least one sensor, at least one display, through which projectionsof the virtual objects are viewed, and at least one processor.

Thereby, it may be preferred that the at least one apparatus worn by theuser allows for binocular or monocular vision to immerses the userwithin the augmented reality environment.

In particular, it could be preferred that the at least one sensorincludes at least one of a digital video camera, an electromagneticsensor, an accelerometer, or a gravimeter.

The user-worn apparatus may be comprised of one or more video sensors(e.g. video cameras) and/or a LIDAR system, an Inertial Measurement Unit(IMU), a Global Positioning System (GPS) sensor, a computer processor,and a see-through MID (Head mounted display) or eyewear. A see-throughHMD may be any medium configured to render an object while stillallowing a user to see through the medium at any location where anobject is not being rendered and view the user's real life environment.The one or more video cameras, IMU, and GPS sensor may generate a dataset used by the computer processor to calculate a pose estimation dataset, wherein the pose estimation data set may determine the pose andlocation of a user, a user's head, and a user-controlled device. Thispose estimation data set enables the user-worn sub-system toappropriately position and occlude synthetic objects in a user's fieldof view, thereby accurately integrating the synthetic objects into theaugmented reality environment.

As well it may be of advantage according to one embodiment of thepresent disclosure that each virtual object can be assigned to anindividual work area having local coordinate space, wherein eachindividual work area is assigned with a trackable marker, wherein thetrackable marker allows the computing devices to identify eachindividual work area independent from their actual location in the realspace by updating the local coordinate space accordingly.

By using a trackable marker, it is possible e.g. to uniquely identify ashelf. The shelf itself could be seen as one certain work area and theremay be virtual objects assigned to physical objects in said shelf. Asone example, a shelf may comprise several storage boxes. If a user linksthe virtual object “take” to one of said boxes, the subsequent user willtake said object. If the shelf is moved and a different shelf is locatedat this position, the subsequent user would grab the wrong tool.According to the present disclosure this problem is solved by the workareas.

The work areas help in particular to define several different types oflocations, like a “tool area” where tools are stored, a general “storagearea” for spare parts, a “repair area” or a “assembly area”. These areasare useful and could as well be pre-defined before the user is settingup the training scenario. Of course it is as well possible that the userdefines said areas himself freely by authoring the scenario.

Said work areas are independent of their physical location, but onlyvirtually linked by the trackable markers. Thereby it may be preferredthat the marker is a data tag, preferably a visual target, in particulara two dimensional or three dimensional barcode, a NFC-Tag and/or aRFID-Tag and/or the marker is an unique identifiable physical objectsensed by the at least one data unit and linked with the virtual objectin the list of objects.

With the help of defining work areas in addition to the linking ofphysical and virtual objects possible unintended failures in use couldbe minimized.

The foregoing aspects and many of the attendant advantages of thisinvention will become more readily appreciated as the same become betterunderstood by reference to the following detailed description, whentaken in conjunction with the accompanying drawing, wherein:

FIG. 1a shows an example of virtual objects displayed to a user forassignment with physical objects; and

FIG. 1b shows a shematic threedimensional view of a visualisation of oneembodiment of a method according to the present disclosure.

The various embodiments will be described in detail with reference tothe accompanying drawings. Wherever possible, the same reference numberswill be used throughout the drawings to refer to the same or like parts.References made to particular examples and implementations are forillustrative purposes, and are not intended to limit the scope of theinvention or the claims.

While illustrative examples are illustrated and described below, it willbe appreciated that various changes can be made therein withoutdeparting from the spirit and scope of the disclosure. In that regard,the detailed description set forth below, in connection with theappended drawings is intended only as a description of various examplesof the disclosed subject matter and is not intended to represent theonly examples. Each example described in this disclosure is providedmerely as an example or illustration and should not be construed aspreferred or advantageous over other examples. The illustrative examplesprovided herein are not intended to be exhaustive or to limit thedisclosure to the precise forms disclosed. Similarly, any stepsdescribed herein may be interchangeable with other steps, orcombinations of steps, in order to achieve the same or substantiallysimilar result.

One example of virtual objects that may be used for the authoring ofscenario cases, in particular of training scenarios, are shown in FIG.1a . With the help of this virtual objects 1 could setting up oftraining scenarios be eased as the same could be placed in the augmentedreality environment directly itself instead of programming the same inadvanced. Examples of such virtual objects may be an info object 3, awarning object 5, a “pick a part” object 7, a “pick a tool” object 9 ora “place a tool” object 11. Of course, several other objects arepossible and can be used within the method according to the presentdisclosure.

In FIG. 1b is a three-dimensional view of one embodiment 20 of thepresent disclosure shown. As can be seen in said FIG. 1b , the user mayassign virtual objects to specified work areas 22, 24 and 26. The firstwork area 22 is defined as a storage area in which physical objects arestored. The first virtual object 28 informs the user to “pic” the objectand “take” the same to the next work area. The subsequent work area 24is a tool location where several different tools are stored. The secondvirtual object 30 informs the user which tool he should “pic” and taketo the third work area 26. Within said work area 26 the third virtualobject 30 informs the user how to mount the object taken from the firstwork area 22 with the tool taken from the second work are 24 to theobject location in the third work area 30.

To set up such a training scenario the user may just place the virtualobjects at the right locations and the next users are these informationshown in the same order as work instructions.

This makes it possible for everyone to set up scenario cases, evenwithout any programming skills.

To do so, it is according to one embodiment of the present disclosurepossible that a user assigns virtual objects to physical objects byphysical actions. The set of virtual objects may comprise functionalelements, in particular visualizations and/or symbols for physicalactions, preferably visualizations and/or symbols for take, use, placeand/or store a physical object, assembly positions, assembly pathways,different types of warnings, tools to be used, general information,preferably a turning moment, inspection tasks and/or how to click aclip.

To link said virtual objects to real physical objects makes it quitesimple to create a training scenario. The user is enable by the methodaccording to the present disclosure to make a physical movement toassign for example a “take” symbol to a certain tool. Afterwards, the“use” symbol is placed above, next to or directly at the physical objectthe grabbed tool should be used. Such steps could easily be completed byinstructions and warnings as well as general information which all canbe linked with physical objects just by a physical movement of the useronly.

The foregoing method descriptions and the process flow diagrams areprovided merely as illustrative examples and are not intended to requireor imply that the steps of the various embodiments must be performed inthe order presented. As will be appreciated by one of skill in the artthe order of steps in the foregoing embodiments may be performed in anyorder. Words such as “thereafter,” “then,” “next,” etc. are not intendedto limit the order of the steps; these words are simply used to guidethe reader through the description of the methods. Further, anyreference to claim elements in the singular, for example, using thearticles “a,” “an” or “the” is not to be construed as limiting theelement to the singular.

The various illustrative logical blocks, modules, circuits, andalgorithm steps described in connection with the embodiments disclosedherein may be implemented as electronic hardware, computer software, orcombinations of both. To clearly illustrate this interchangeability ofhardware and software, various illustrative components, blocks, modules,circuits, and steps have been described above generally in terms oftheir functionality. Whether such functionality is implemented ashardware or software depends upon the particular application and designconstraints imposed on the overall system. Skilled artisans mayimplement the described functionality in varying ways for eachparticular application, but such implementation decisions should not beinterpreted as causing a departure from the scope of the presentinvention.

The hardware used to implement the various illustrative logics, logicalblocks, modules, and circuits described in connection with the aspectsdisclosed herein may be implemented or performed with a general purposeprocessor, a digital signal processor (DSP), an application specificintegrated circuit (ASIC), a field programmable gate array (FPGA) orother programmable logic device, discrete gate or transistor logic,discrete hardware components, or any combination thereof designed toperform the functions described herein. A general-purpose processor maybe a microprocessor, but, in the alternative, the processor may be anyconventional processor, controller, microcontroller, or state machine. Aprocessor may also be implemented as a combination of computing devices,e.g., a combination of a DSP and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with a DSPcore, or any other such configuration. Alternatively, some steps ormethods may be performed by circuitry that is specific to a givenfunction.

In one or more exemplary aspects, the functions described may beimplemented in hardware, software, firmware, or any combination thereof.If implemented in software, the functions may be stored as one or moreinstructions or code on a non-transitory computer-readable medium ornon-transitory processor-readable medium. The steps of a method oralgorithm disclosed herein may be embodied in a processor-executablesoftware module which may reside on a non-transitory computer-readableor processor-readable storage medium. Non-transitory computer-readableor processor-readable storage media may be any storage media that may beaccessed by a computer or a processor. By way of example but notlimitation, such non-transitory computer-readable or processor-readablemedia may include RAM, ROM, EEPROM, FLASH memory, CD-ROM or otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium that may be used to store desired programcode in the form of instructions or data structures and that may beaccessed by a computer. Disk and disc, as used herein, includes compactdisc (CD), laser disc, optical disc, digital versatile disc (DVD),floppy disk, and blu-ray disc where disks usually reproduce datamagnetically, while discs reproduce data optically with lasers.Combinations of the above are also included within the scope ofnon-transitory computer-readable and processor-readable media.Additionally, the operations of a method or algorithm may reside as oneor any combination or set of codes and/or instructions on anon-transitory processor-readable medium and/or computer-readablemedium, which may be incorporated into a computer program product.

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. The sole and exclusive indicator of the scope of the invention,and what is intended by the applicants to be the scope of the invention,is the literal and equivalent scope of the set of claims that issue fromthis application, in the specific form in which such claims issue,including any subsequent correction.

1. A method comprising: establishing, using at least one computingdevice, an augmented reality environment; wherein the augmented realityenvironment is comprised of a real space in which a user is physicallylocated; wherein the augmented reality environment comprises a set ofvirtual objects projected into the real space assignable to physicalobjects by the user; sensing, by at least one sensor, a particularphysical action of the user in the real space in which the user isphysically located representative for the assignment of the virtualobject to the physical object; in response to sensing the particularphysical action in the real space in which the user is physicallylocated, updating, by at least one application executing on the at leastone computing device, a set of one or more properties of a list linkingthe physical and the virtual objects based on the particular physicalaction sensed by the at least one sensor, and wherein updating the setof one or more properties of the list linking the physical and thevirtual objects based on the particular physical action sensed by the atleast one sensor causes a subsequent user to see in the establishedaugmented reality the virtual objects assigned by the user to thephysical objects.
 2. The method according to claim 1, wherein thesubsequent user is the first user interacting with the augmented realityenvironment after having assigned at least one of one virtual object tothe physical object comprising the updated list and a new user notidentical with the first user.
 3. The method according to claim 1,wherein updating the set of one or more properties of the list linkingthe physical and the virtual objects based on the particular physicalaction sensed by the at least one sensor causes the comparing of thesensed physical actions of a subsequent user with the predeterminedactions to be done by the subsequent user defined by the assignedvirtual objects to the physical objects by the first user, whereinsimilarities and differences of the physical actions of the subsequentuser to the defined actions by the first user are sensed by the at leastone sensor and compared by the at least one computing device, wherein inparticular the sensor is configured and adapted to detect at least onebody movement comprising hand movements, viewing at positions,eyetracking of the subsequent user and/or the use of interactiondevices, in particular the use of a joystick or a controller by thesubsequent user, wherein in particular a user-worn or subsequentuser-worn apparatus comprise of one or more video sensors and a LIDARsystem as a sensor, an Inertial Measurement Unit (IMU) as a sensor, aGlobal Positioning System (GPS) sensor, a computer processor, and asee-through MID (Head mounted display) and eyewear as the sensor.
 4. Themethod according to claim 3, wherein, the level of detailing of thevirtual objects assigned to the physical objects shown to a subsequentuser depends on a level of experience of the subsequent user, determinedby the monitored similarities and differences of executed actions andpredetermined actions defined by the assigned virtual objects, whereinin case of more differences the level of detailing becomes higher and incase of more similarities the level of detailing becomes lower.
 5. Themethod according to claim 1, wherein the set of virtual objectscomprises functional elements, in the form of one or more ofvisualizations and symbols for physical actions, wherein the functionalelements are for at least one of take, use, place and store a physicalobject, assembly positions, assembly pathways, different types ofwarnings, tools to be used, general information, in the form of one ormore of a turning moment, inspection tasks and how to click a clip. 6.The method according to claim 1, wherein the list linking the physicaland the virtual objects based on the particular physical action isgenerated in the order the physical actions are performed by the userand shown in said order determined by the first user to the subsequentusers.
 7. The method according to claim 1, wherein the particularphysical action by the at least one user is comprised of at least of onebody movement, direct input of data into the at least one computerdevice, and at least one vocalization.
 8. The method according to claim6, wherein at least one body movement comprises at least one of handmovements, viewing at positions, eyetracking and the use of interactiondevices in the form of at least one of a joystick and a controller. 9.The method according to claim 1, wherein the particular physical actionscomprises a predefined component to be executed by the user to indicatethe computing device establishing the augmented reality to assign avirtual object to a physical object.
 10. The method according to one ofthe claim 1, wherein the particular physical actions to assign a virtualobject to a physical object is determined by the at least one computingdevice automatically and no predefined movement of the user is needed,wherein the assignment of the virtual to the physical object isdetermined by the at least one computing device by at least one ofmachine learning and artificial intelligence.
 11. The method accordingto claim 1, wherein the user views the augmented reality environment bywearing at least one apparatus that projects virtual objects into thereal space in which the user is physically located; wherein the at leastone apparatus includes the at least one sensor, at least one displaythrough which projections of the virtual objects are viewed, and atleast one processor.
 12. The method according to claim 11, wherein theat least one apparatus worn by the user allows for binocular vision toimmerses the user within the augmented reality environment.
 13. Themethod according to claim 1, wherein the at least one sensor includes atleast one of a digital video camera, an electromagnetic sensor, anaccelerometer, or a gravimeter.
 14. The method according to claim 1,wherein each virtual object can be assigned to an individual work areahaving local coordinate space, wherein each individual work area isassigned with a trackable marker, wherein the trackable marker allowsthe computing devices to identify each individual work area independentfrom their actual location in the real space by updating the localcoordinate space accordingly.
 15. The method according to claim 14,wherein the marker is a data tag, in the form of a visual target thathas at least one of, a two dimensional or three dimensional barcode, aNFC-Tag, a RFID-Tag, and the marker is an unique identifiable physicalobject sensed by the at least one data unit and linked with the virtualobject in the list of objects.